feat: enhance file download and caching mechanisms by egalvis27 · Pull Request #404 · internxt/drive-desktop-linux

egalvis27 · 2026-06-29T20:34:01Z

What is Changed / Added

Fixed invalid EOF and out-of-bounds range handling in virtual drive hydration and ranged downloads.
Added guards to skip zero-length and invalid block prefetch/download work.
Extended block prefetching from thumbnail reads to normal file reads used for open/copy flows.
Added environment-based tuning for normal read prefetch size.
Unified duplicated prefetch constants in the read callback path.
Removed noisy success metrics from ranged download logs and kept failure warnings only.
Added and updated regression tests for hydration bounds, prefetch behavior, and download range handling.

Why

Prevent crashes caused by negative range lengths and invalid buffer allocations near end-of-file.
Stop wasted work and memory pressure from scheduling blocks outside real file bounds.
Improve perceived file open and copy performance, not only thumbnail loading.
Make prefetch behavior easier to tune without editing code.
Reduce small code duplication and keep prefetch defaults consistent.
Keep logs useful during failures without adding per-block noise during normal operation.
Lock behavior down with focused tests so performance changes do not reintroduce correctness bugs.

…mance

sonarqubecloud · 2026-06-29T20:38:28Z

Quality Gate failed

Failed conditions
75.4% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

AlexisMora · 2026-07-07T12:25:18Z

+const ALLOWED_BLOCK_SIZE_MB = new Set([1, 2, 4, 8]);
+
+function getConfiguredBlockSizeInMb() {
+  const configuredValue = process.env.INTERNXT_DRIVE_DOWNLOAD_BLOCK_SIZE_MB;


Why this needs to be a enviroment variable

My idea is that these values should be easy to change. Right now, the optimal settings are 4 MB per download and 3 prefetch blocks. These values can be increased if changes are made to the backend to reduce the latency of download requests, which would greatly improve performance.

Looking back, we can adjust these values to achieve better performance if the conditions are right; that’s why I set them as environment variables so they can be easily modified.

AlexisMora · 2026-07-07T12:26:58Z

  blockLength,
 }: Props): Promise<Result<void, Error>> {
  if (isAborted(state)) return { data: undefined };
+  if (blockLength <= 0 || blockStart >= virtualFile.size) return { data: undefined };


Is this something that can happen? meaning: Fuse can ask for a block that is negative?

It's not just to validate negative ranges; it's also to prevent readings in the 0 range—which fuse does sometimes—and to ensure that invalid ranges aren't attempted, which is rare but does happen.

AlexisMora · 2026-07-07T12:29:24Z

+    return { blockStart: Math.max(0, Math.min(range.position, fileSize)), blockLength: 0 };
+  }
+
+  const clampedStart = Math.max(0, range.position);


Why? i assume that range will always be 0 or more

We might assume that's the case, but there are very rare instances where it might not be; this change is a simple safeguard that definitively prevents that from happening.

AlexisMora · 2026-07-07T12:33:49Z

 import { type HandleReadDeps, type ReadRange } from './types';
 import { isThumbnailProcess } from './thumbnail-processes';
+
+const PREFETCH_DEFAULT_BLOCKS_AHEAD = 5;


shouldn this be on constants?

You're right, I've moved it.

AlexisMora · 2026-07-07T12:42:40Z

+function getThumbnailPrefetchBlocksAhead() {
+  return getPrefetchBlocksAhead({
+    configuredValue: process.env.INTERNXT_DRIVE_THUMBNAIL_PREFETCH_BLOCKS_AHEAD,
+  });
+}
+
+function getReadPrefetchBlocksAhead() {
+  return getPrefetchBlocksAhead({
+    configuredValue: process.env.INTERNXT_DRIVE_READ_PREFETCH_BLOCKS_AHEAD,
+  });
+}


As I mentioned earlier, the idea is to be able to easily change the values.

AlexisMora · 2026-07-07T12:44:13Z

+  return Math.max(0, Math.min(parsed, PREFETCH_MAX_BLOCKS_AHEAD));
+}
+
+function getThumbnailPrefetchBlocksAhead() {


Why, for a thumbnail, we need to do a prefetch? would not this slow the initial load of the file explorer?

I've been running tests focused on thumbnails, and you're right—the improvement is very small, and it slows down the network for other processes.

AlexisMora · 2026-07-07T12:45:13Z

  }
 }

+function schedulePrefetchBlocksAhead({


Shouldnt we test this separtedly since it has a lot of logic?

AlexisMora · 2026-07-07T12:47:35Z

 }): Promise<Result<void, Error>> {
  const { blockStart, blockLength } = expandToBlockBoundaries({ range, fileSize: virtualFile.size });

+  if (blockLength <= 0 || blockStart >= virtualFile.size) {


Again, if this is a realistic scenario, why not prevent from calling this whole functionality?

This security check has been deliberately included as a protective measure.
Even though we attempt to filter out such cases beforehand, this path can still be reached in realistic extreme scenarios (reads at the end of the file, zero-length reads, obsolete ranges following changes in file size).

AlexisMora · 2026-07-07T12:49:28Z

    const downloads = missingBlocks.map((block) => {
      const start = block * BLOCK_SIZE;
      const end = Math.min(start + BLOCK_SIZE, virtualFile.size);
+      const boundedBlockLength = end - start;


Same, i feel like there has to be a way to prevent this whole "negative block request" instead of just adding everywhere the check, this just adds unnecessary complexity dont you think

You're right—the extra validation was too conservative; the first validation at the beginning of the function is enough.

AlexisMora · 2026-07-07T12:50:30Z

+        if (
+          !shouldEmitProgress({
+            now: Date.now(),
+            bytesDownloaded,
+            fileSize,
+            state: progressReporterState,
+          })
+        ) {
+          return;
+        }
+


Why is this necessary?

This is just to avoid spamming progress updates.
Downloads can report progress very frequently, and without this check we would send too many UI updates for very small changes.
So this acts like a small throttle: it only emits from time to time, but still allows the final update through when the download reaches 100%.

AlexisMora · 2026-07-07T12:51:07Z

+const PROGRESS_UPDATE_INTERVAL_MS = 250;
+
+type ProgressReporterState = {
+  lastUpdateAt: number;
+};
+
+function shouldEmitProgress({
+  now,
+  bytesDownloaded,
+  fileSize,
+  state,
+}: {
+  now: number;
+  bytesDownloaded: number;
+  fileSize: number;
+  state: ProgressReporterState;
+}) {
+  const reachedEnd = bytesDownloaded >= fileSize;
+  const elapsedSinceLastUpdate = now - state.lastUpdateAt;
+
+  if (!reachedEnd && elapsedSinceLastUpdate < PROGRESS_UPDATE_INTERVAL_MS) {
+    return false;
+  }
+
+  state.lastUpdateAt = now;
+  return true;
+}
+


IF this is necessary this should be in its own file, where we should test it separatedly

AlexisMora · 2026-07-07T12:59:38Z

+    let bytesRead = 0;
+    let buffer = Buffer.alloc(length);
+
+    response.data.on('data', (chunk: Uint8Array) => {
+      const source = Buffer.from(chunk);
+      const requiredLength = bytesRead + source.length;
+
+      if (requiredLength > buffer.length) {
+        const next = Buffer.alloc(Math.max(buffer.length * 2, requiredLength));
+        buffer.copy(next, 0, 0, bytesRead);
+        buffer = next;
+      }
+
+      source.copy(buffer, bytesRead);
+      bytesRead += source.length;
+    });


Why do we need this?

We need to do this because the HTTP response comes back as a stream, not as a complete Buffer.
So before returning, we have to collect all the chunks and build the final contiguous buffer.
If we returned earlier, we could end up decrypting or processing incomplete data.

…gress reporting

coderabbitai · 2026-07-07T21:13:00Z

Warning

Review limit reached

@egalvis27, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 57 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ec2399fe-2beb-4e24-824d-a315e4381942

📥 Commits

Reviewing files that changed from the base of the PR and between e7f872c and 3d88862.

📒 Files selected for processing (27)

.env.example
src/backend/features/fuse/on-read/download-cache/constants.ts
src/backend/features/fuse/on-read/download-cache/download-and-save-block.ts
src/backend/features/fuse/on-read/download-cache/expand-to-block-boundaries.test.ts
src/backend/features/fuse/on-read/download-cache/expand-to-block-boundaries.ts
src/backend/features/fuse/on-read/download-cache/hydration-state.test.ts
src/backend/features/fuse/on-read/download-cache/hydration-state.ts
src/backend/features/fuse/on-read/handle-read-callback.test.ts
src/backend/features/fuse/on-read/handle-read-callback.ts
src/backend/features/fuse/on-read/read-or-hydrate.test.ts
src/backend/features/fuse/on-read/read-or-hydrate.ts
src/backend/features/virtual-drive/services/operations/read.service.ts
src/backend/features/virtual-drive/services/operations/should-emit-progress.test.ts
src/backend/features/virtual-drive/services/operations/should-emit-progress.ts
src/context/storage/StorageFiles/application/download/download-with-progress-tracking.test.ts
src/context/storage/StorageFiles/application/download/download-with-progress-tracking.ts
src/infra/drive-server/client/interceptors/rate-limiter/attach-rate-limiter-interceptors.test.ts
src/infra/drive-server/client/interceptors/rate-limiter/attach-rate-limiter-interceptors.ts
src/infra/drive-server/client/interceptors/rate-limiter/create-request-interceptor.test.ts
src/infra/drive-server/client/interceptors/rate-limiter/create-request-interceptor.ts
src/infra/drive-server/client/interceptors/rate-limiter/create-response-interceptor.test.ts
src/infra/drive-server/client/interceptors/rate-limiter/create-response-interceptor.ts
src/infra/drive-server/client/interceptors/rate-limiter/rate-limiter.types.ts
src/infra/drive-server/client/interceptors/rate-limiter/update-state-from-headers.test.ts
src/infra/drive-server/client/interceptors/rate-limiter/update-state-from-headers.ts
src/infra/environment/download-file/download-file.test.ts
src/infra/environment/download-file/download-file.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/improve-download-speed

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

egalvis27 added 2 commits June 29, 2026 15:28

feat: enhance file download and caching mechanisms

1f8371d

feat: increase default block size to 4MB for improved download perfor…

481a78d

…mance

AlexisMora reviewed Jul 7, 2026

View reviewed changes

AlexisMora requested changes Jul 7, 2026

View reviewed changes

egalvis27 requested a review from AlexisMora July 7, 2026 21:05

feat: implement prefetch block configuration and enhance download pro…

3d88862

…gress reporting

Uh oh!

Conversation

egalvis27 commented Jun 29, 2026

What is Changed / Added

Why

Uh oh!

sonarqubecloud Bot commented Jun 29, 2026

Quality Gate failed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexisMora Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jul 7, 2026

Review limit reached

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AlexisMora Jul 7, 2026 •

edited

Loading